In the field of remote observation of objects, particularly during the course of military reconnaissance, it is important to track moving objects, for example launched rockets having firing engines, over a great distance of up to 1500 km and to at first detect rockets that have stopped firing, with sufficient, if applicable artificial illumination, over a distance of up to 1000 km, to track them along their movement path, and to identify them. Even if optical observation apparatuses having long focal length lenses and high-resolution image sensors are used, it is not possible to obtain high-resolution images reproducing the engine jet flame object with sharp focus within the range of a few meters at an object distance of more than 500 km to as much as 1500 km, because the brightness distribution within the jet flame fluctuates in all directions, at high frequency, in a range of up to 100 m. In spite of such reduced imaging performance, however, it must be possible to determine whether the detected object is a military or civilian launch vehicle or a decoy. Furthermore, it must be possible for object recognition to take place over a sufficient period of time, with a reliable result, at a precision of a few meters, in order to be able to measure the flight path of the object (for example, a rocket) from this process with such precision that stable tracking of the observation apparatus is possible, and that combating the flying object on its flight path can take place.
From the general state of the art, large X-band radar systems are known for such target tracking purposes, which systems must be stationed closely enough along an expected rocket route that they constantly have a flying rocket in their viewing field above the horizon. This not only requires great effort and expenditure, but is frequently also not sufficiently possible for political reasons. Furthermore, such radar stations can only determine the position of target objects and measure their radar reflection cross-section at a distance of 1000 km crosswise to the viewing direction, with a precision of several kilometers, but cannot undertake any precise identification. Decoys, in particular, can usually not be distinguished from true warheads by means of radar, and this causes great problems.
Furthermore, satellites in geostationary orbits are known from the state of the art, which can discover launched rockets in the medium infrared range, using telescopes. Because the discovery from above must take place against the warm earth background, with a great number of false targets that are difficult to recognize, these systems must battle against comparatively low sensitivity. On the basis of the observation from above, the sensor also sees only the less bright and fluctuating part of the engine flame. As a result, its measurement accuracy is limited to a few hundred meters, and cannot be significantly improved due to system limitations.
A camera system for detecting and tracking the path of moving objects situated at a great distance, from a high-flying aircraft that flies above the dense atmosphere, is known from German patent document DE 10 2011 010 337, which is not a prior publication.
This system has the great advantage of being able to observe the launched rockets practically outside the atmosphere, from below, against the background of cold outer space, and does not need to suppress any false targets, except for stars and close heavenly bodies, the position of which is precisely known, in order to prevent false alarms.
A further great advantage of observation from below and to the side of the flight path is that from there, the viewing direction onto the hot core region of the engine jet, which has a temperature above 2000° K, is clear directly at the jet outlet. This core region of the jet has a light density that is one hundred times greater than the jet farther away, and is firmly fixed in place at the jet outlet, in other words does not perform any fluctuations. As a result, an extremely bright (1 megawatt/m2) light source point, a few meters in diameter, is available for tracking the flight path of the rocket, precisely and constantly.
Exemplary embodiments of the present invention are directed to a sensor that can locate this point light source over a distance of up to 1500 km, with a precision of a few meters, and follow it.
This can be achieved using a multi-spectral camera system for the near infrared range, which is detailed below. The multi-spectral camera can record multi-spectral images (e.g. at 700 nm, 800 nm, and 950 nm) of a scene sequentially, using a motor-driven filter wheel having at least three narrow-band (e.g. 20 nm) transmission filters. From this, a temperature image of the scene, with a resolution of 50° K, for example, can be calculated by means of conversion according to the black-body radiation laws. At this resolution, according to the invention, the hot core region of a solid-fuel rocket, having a temperature of approximately 2300° K, and its characteristic shape can be clearly differentiated from the hot core region of a liquid-fuel rocket, having a temperature of 2100° K, and a different shape, and with sufficient optical resolution of the camera from 1 m to 2 m, at a distance of 1000 km, the size of the core region and its temperature distribution can also be measured. With these data, military solid-fuel rockets can be differentiated from civilian liquid-fuel rockets, and different rocket types can be differentiated by the size, number, and arrangement of the engines.
This camera system has a camera provided with a lens having a long focal length, which camera is disposed on a position-stabilized platform. This camera is provided with a high-speed shutter as well as a first and a second image sensor. The light radiation captured by the camera lens can be optionally guided to the first or the second image sensor, with a further telephoto lens being assigned to one of the image sensors. The camera optics furthermore have a pivoting mirror, with which it is possible to scan a field line by line, by means of pivoting the mirror, with the captured image signal being passed to one of the image sensors. If a target object is recognized during this scanning process, the light beam is deflected to the other image sensor, which is then used for object identification and, if applicable, for target tracking.
Exemplary embodiments of the present invention are directed to a method for image processing, with which it is possible to process image data collected, even over great distances, for example several hundred kilometers, particularly at a distance of 100 km to 500 km, in such a manner that it is possible to recognize an object contained in the recorded scene by means of these processed image data, by means of the processed image data. Exemplary embodiments of the present invention are also directed to performing automatic object recognition using this method of image processing. Finally, exemplary embodiments of the present invention are directed to observation apparatuses with which these methods can be implemented.
An exemplary method according to the invention, equipped in this manner, has the following method steps:
a) collecting image data of a scene as electromagnetic radiation, such as light in the visible spectrum, in the infrared spectrum, or in the ultraviolet spectrum, for example, by means of an optical device;
b) processing the image data obtained in step a) by means of image processing, to improve the signal-to-noise ratio of the image data, wherein the processing is carried out in the following partial steps:
b1) dividing a raw image that contains the image data into lines and columns, to create a raster image;
b2) superimposing a central raster filter element of a raster filter having an odd number of lines and an odd number of columns onto a raster image element;
b3) determining the brightness values of each of the raster image elements covered by the raster filter, wherein except for the central raster filter element, every other raster filter element has an individual light-reducing property;
b4) adding up the brightness values determined in step b3) to produce a total brightness value, and assigning this total brightness value to the raster image element covered by the central raster filter element;
b5) repeating steps b2) to b4) for all remaining raster image elements;
c) producing a result image having the same resolution as the raw image from the total brightness values of the raster image elements obtained in step b).
By means of this image processing method according to the invention, the signal-to-noise ratio of the collected raw image is improved, in that the brightness progression of the raw image is filtered. Furthermore, noise pixels that emerge from the background of the raw image are removed, and thereby the image is also filtered. The brightness progression in the result image obtained is constant and can be differentiated, as compared with the raw image, and the image contrast is improved, so that an object contained in the raw image stands out more distinctly and clearly in the result image.
A preferred further development of this method according to the invention for image processing is characterized in that in step a), the image data of the scene are collected in more than one electromagnetic wavelength range, in order to thereby obtain raw images of the scene in different spectral ranges; that steps b) and c) are performed for all the raw images of the scene, in order to obtain result images of different spectral ranges, and that the result images of the different spectral ranges are combined to form a multi-spectral result image by means of superimposition.
Recording the scene in different spectral ranges by means of sequentially recording images using narrow-band (e.g. 20 nm) filters and combining the raw images of these different spectral ranges, processed according to the invention, in each instance, to form a multi-spectral result image, improves the informational value of the image result obtained.
This is particularly the case if the individual result images of different spectral ranges are combined in monochrome manner, colored with different colors, to produce the multi-spectral result image. A multi-spectral image having filters selected to match the temperature of the body being observed (e.g. 2300° K) on the short-wave flank of the black-body radiation curve can be used to convert the multi-spectral color image to a temperature image. This temperature image makes it possible to find a small, stable temperature region within a significantly larger, possibly also locally brighter, greatly fluctuating background brightness field, such as, for example, a rocket jet tail.
It is advantageous, in this connection, if collecting the image data of the scene in the different spectral ranges takes place using different spectral range filters, in rapid sequence, in terms of time, by means of a high-speed camera. As a result, it becomes possible to record almost time-synchronous raw images of the scene, in comparison with the movement of the object being observed, in different spectral ranges, which images differ only insignificantly with regard to the position of the object in the image, because of the extremely short time sequence of the respective recorded images, so that these recorded images can easily be combined by means of superimposition to produce the multi-spectral result image.
A method for automatic object recognition in accordance with exemplary embodiments of the present invention involves an image processing method in which:                recording the scene in step a) is performed at different angles of rotation about the optical axis of the optical device;        for every angle of rotation, a result image is produced according to the steps of the method according to the invention for image processing;        the individual result images are compared with sample images of individual objects stored in an object database; and        each sample image having the least deviation from one or more of the result images identifies the object contained in the scene and determines the position of the object in the result image.        
By means of this automatic object recognition method, automatic object identification is made possible, using the image processing method according to the invention. Furthermore, the position of the object in the result image can be determined by means of this automatic object recognition method, and thereby a direction vector of the movement of the object (for example, a rocket) can already be predicted with greater accuracy than according to the state of the art, in the case of a single recorded and analyzed scene.
A preferred further aspect of this object recognition method according to the invention is characterized in that determining the position of the object in the result image takes place by means of determining the raster elements of the result image that agree with corresponding raster elements of the sample image.
Exemplary embodiments of the present invention are directed to an observation apparatus for image processing and an observation apparatus configured for carrying out the method according to the invention for automatic object recognition, using the method according to the invention for image processing.
In the two observation apparatuses according to the invention, an embodiment in which the image processing device has an image rasterization module and a raster filter module is advantageous.
In a first variant of the observation apparatus according to the invention, the image rasterization module has a matrix-like arrangement of light guide elements, which are disposed between the optical device and a sensor sensitive to the detected radiation. In this connection, at least part of the light guide elements have a brightness-reducing raster filter element of the raster filter module assigned to them, in each instance. The optical device is configured in such a manner that it depicts the collected image data as a raw image in an entry plane of the image rasterization module, and it is furthermore configured in such a manner that the raw image can be displaced on the entry plane with reference to an entry plane of the image rasterization module. Furthermore, a computer unit is provided, which receives a brightness signal from the sensor, and on which software runs, which implements method step c) as well as other method steps. This advantageous embodiment of the observation apparatus implements the method steps according to the invention in optical-mechanical manner.
When “brightness” is mentioned in this document, the term is not restricted to the spectrum of visible light, but rather also comprises the intensity of radiation in a non-visible spectrum, such as, for example, in the infrared spectrum or in the ultraviolet spectrum, but without being restricted to these.
Alternatively, the method steps according to the invention can also be implemented in software, for which purpose the observation apparatus suitable for this purpose is characterized in that the optical device is followed by an image sensor, that the optical device is configured in such a manner that it depicts the collected image data in a sensor plane of the image sensor, that a computer unit is provided, which receives an image signal from the image sensor, and that software runs in the computer unit, which software implements method steps b) and c), as well as other method steps, wherein the image rasterization module and the raster filter module are configured as a subroutine of the software.
This advantageous embodiment of the observation apparatus implements the method steps according to the invention in optical-electronic manner.
The combined multi-spectral images are put together in the computer unit only after processing of the individual raw images, from a greater number of superimposed processed individual images, which have been recorded in different spectral colors. The combined multi-spectral image then possesses a much better signal-to-noise ratio than the raw images, of preferably above 100, if the number of superimposed individual images is sufficient, by means of averaging over the many individual images.
The combined multi-spectral images are preferably evaluated using a multi-spectral image evaluation and identification method according to FIG. 3 or FIG. 4, in the image evaluation device 25, 125. In this connection, preferably first an observation of the target behavior is undertaken, and, in particular, the number and the path curves of the visible objects are determined. Then a file of all the flight objects and their flight paths is compiled, which permits reliable recognition of all the objects during subsequent measurements and allows extrapolation of their flight paths in the future, particularly calculation of possible strike points of the flight object at the end of the flight path. Furthermore, the behavior of the objects (separation of a rocket stage, ejection of decoys, flight maneuver of a warhead) can be observed and analyzed as a time progression.
The combined multi-spectral images are furthermore subjected, in multi-spectral target image recognition according to FIG. 3 or FIG. 4, to a comparison with a reference target image database that is incorporated and stored in the memory device 29, 129. In this way, the target image recognition can recognize imaged targets as target objects of a specific type, and consequently identify them. The image recognition can work more reliably and with more precise distinction if the added multi-spectral images are subjected to processing in multiple states before they are worked on.
For this purpose, the combined multi-spectral images are first converted to a standardized form, according to the invention, in that at first, the orthogonally vectorially added total brightness is formed for each pixel as the brightness value, and subsequently all the color components are standardized with the total brightness. The total color vector then consists of the brightness and the standardized color values. In this way, a color coordinate system having any desired number of spectral components can be defined, and in this system all the color operations, which are defined only in three colors in the RGB system, can be carried out multi-spectrally.
For all the color components and the brightness of each image pixel, the image processing performs an operation on a digital basis, according to FIG. 2, during which operation the image is simultaneously filtered and made differentiable, in which interference pixels that are still present after averaging by way of multiple images are removed and filtered out, and in which brightness transitions and edges are accentuated, so that the result image becomes sharper, more contrast-rich, and clearer in terms of colors, and can be evaluated more reliably. This is achieved by means of transformation of all the color components and of the brightness component of the standardized image, with the filter matrix of the raster filter 122, which has a size of 5×5 pixels in the example shown, in the apparatus for processing images having a poor signal-to-noise ratio, on a digital basis, according to FIG. 2.
Preferably, the image is also subjected to affine color transformation, in which spectral components that characterize the target are stretched and therefore become easier to evaluate, and non-typical spectral components are compressed. This allows greater separation clarity between true targets and false targets that could be mistaken for true targets, in the case of correlation of the result images with the true target objects hidden in the images, in the image recognition, than without this image processing.
The multi-spectral image recognition can be carried out either using the apparatus for multi-spectral image recognition on an optical basis according to FIG. 3 or, with greater precision, using the apparatus for multi-spectral image recognition on a digital basis according to FIG. 4.
In the case of the apparatus on an optical basis (according to FIG. 3), the telescope 110 produces a real target image of a remote target, having a size of 25×25 pixels, by way of the deflection mirror 112, in the plane 121 of the front surface of the optical 5×5 light-guide bundle of the image recording device 120, which takes up the same surface area as 5×5 pixels of the real target image. The scanning mirror 112 deflects the target image horizontally and vertically, in such a manner that each center pixel of each 5×5 pixel block passes over the center light-guide element of the 5×5 optical light-guide bundle sequentially. The twenty-five values for the 5×5 pixel blocks of each image having a size of 25×25 pixels are stored in the computer unit 126 for all the spectral ranges. This is repeated for twelve rotational positions, over 360° of the 25×25 pixel image. Search ranges of 15×15 pixel blocks from the target images are compared with the reference images being search, having a size of 15×15 pixels, for the value of each center pixel of each 5×5 pixel block, where the differences of the nine coefficient values of input image search range and current reference image, in each instance, are formed. The position and the rotational position at which the smallest difference amount occurs, and at which this amount goes below a predetermined minimal value, is registered as the position and rotational position of a found target of the current reference image class. Image parts that project out of the image field are not taken into consideration. The position resolution of the image recognition amounts to five pixels horizontally and vertically, in this connection.
In the case of the apparatus on a digital basis (FIG. 4), the telescope 110 produces a real target image of a remote, illuminated target having a size of 25×25 pixels, by way of the deflection mirror 112, in the image plane of the image recording device 120 that has an NIR (near-infrared) camera, for example. The camera converts the light signal to a digital multi-spectral image with high resolution. For the multi-spectral image recognition, characteristic values of an evaluation function according to the invention are calculated for every search pixel position in the search image (size 25×25 pixels), as described above. By means of the described formation of the evaluation function, the number of rotational positions to be examined can be limited to twelve, without any loss of separation sharpness.
Using a near-infrared sensor system, which is configured according to one of the aforementioned examples, target detection, flight path tracking, flight path measurement, and target observation and target identification of launched rockets, even after engine shutoff, can be carried out at distances up to 500 km.
Preferred exemplary embodiments of the invention, with additional configuration details and further advantages, will be described in greater detail and explained below, making reference to the attached drawings.